Multi-objective multiagent credit assignment in reinforcement learning and NSGA-II

Multiagent systems have had a powerful impact on the real world. Many of the systems it studies (air traffic, satellite coordination, rover exploration) are inherently multi-objective, but they are often treated as single-objective problems within the research. A key concept within multiagent systems is that of credit assignment: quantifying an individual agent’s impact on the overall system performance. In this work we extend the concept of credit assignment into multi-objective problems, broadening the traditional multiagent learning framework to account for multiple objectives. We apply credit assignment through difference evaluations to two different policy selection paradigms to demonstrate the broad applicability of the proposed approach. We first examine reinforcement learning, in which we improve performance by (i) increasing learning speed by up to 10x (ii) reducing sensitivity to unmodeled disturbances by up to 98.4% and (iii) producing solutions that dominate all solutions discovered by a traditional team-based credit assignment schema. We then examine a state-of-the-art multi-objective evolutionary algorithm, NSGA-II. We derive multiple methods for incorporating difference evaluations into the NSGA-II framework. Median performance of the NSGA-II considering credit assignment dominates best-case performance of NSGA-II not considering credit assignment in a multiagent multiobjective problem. Our results strongly suggest that in a multiagent multi-objective problem, proper credit assignment is at least as important to performance as the choice of multi-objective algorithm. This work was partially supported by the National Energy Technology Laboratory under grant DEFE0012302. Logan Yliniemi University of Nevada, Reno E-mail: [email protected] Kagan Tumer Oregon State University E-mail: [email protected] 2 Logan Yliniemi, Kagan Tumer